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ABSTRACT 



This paper reviews an earlier study examining the variables 
affecting the success of distance learning programs. In 1995, Dr. Ellen 
Wagner developed a list identifying the following 16 variables which need to 
be considered if a distance learning program is to be successful: needs 
assessment; audience analysis; instructional design; course selection; course 
reconfiguration; lesson planning; interactive instructional strategies; 
instructional delivery systems; adapting media and materials for distance 
delivery; developing effective graphics; using facilitators at distance 
education sites; learner and instructor support services; program evaluation 
and learner assessment; organized readiness; inter- institutional 
partnerships; and regulatory environment. Nine case studies and three 
non-case study journal articles that describe distance learning activities 
and/or success factors were reviewed to test the validity of these variables. 
Findings suggest that these variables are generally considered during course 
design or implementation. The suitability of the factors, however, is 
dependent on the nature of the organization into which the program is being 
delivered, and the selection of the delivery mode. Three tables present study 
results. ( AEF) 
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Validating factors that impact on the 
success of distance learning initiatives 



Introduction 

Distance learning has become big business. In recent years, it has grown beyond being a 
sub-set of the training and education industry into an industry which stands alone in its 
own right. But much is still unknown about the industry and the opportunities it brings to 
an organization. 

Research is scarce in the distance learning world. Moore and Kearsley (1996) suggest 
that the time is right to fill this research void (pp. 211-212). In this theoretical vacuum, 
however, many organizations are turning to distance learning to give themselves a 
competitive advantage in the global market place. The success of these endeavors has 
been mixed and many of these successes have been recorded in the literature in the form 
of case studies. 

But why have some distance learning programs succeeded while others have faltered? 
What is it that distinguishes a successful distance learning program from an unsuccessful 
one? If suitable answers to these questions can be found, organizations that follow the 
advice stand to gain a considerable advantage over their competitors. The number of 
variables that affect the success of distance learning programs is considerable, however, 
identifying them is the first step in the pursuit of answers to these questions. 

While a large number of case studies are identifiable in the literature, there are far fewer 
research efforts which seek to identify the factors which are critical to the success of 
distance learning programs. The reason for this, it is suggested, is that it is very difficult 
to isolate the primary variables upon which success depends. Nevertheless, studies have 
been conducted which aim to improve the likelihood of success through the identification 
of such factors. 
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Given that the number of variables that affect distance learning are so varied, the 
usefulness of this type of research must be questioned. Are there too many variables, 
settings and types of media which can impact on the success of a program? Would the 
research efforts be better spent in other research areas such as policy, marketing or 
learner support systems? 

The purpose of this paper is to review an earlier study which aims to identify the 
variables affecting the success of distance learning programs in an attempt to validate the 
usefulness this type of research. 

It is suggested that this research is in an embryonic state and that a start must be made in 
order to realize the full potential of distance learning projects. As a result, the value of 
research into identifying critical success factors is worthy of the investment. If the 
success rate of distance learning programs can be improved, other areas will also 
improve. Marketing will become easier; investment risks will decrease and organization 
motivation for this agent of change will increase. This can only have a positive effect on 
the industry. 



Variables in distance learning 

In 1993, Dr Ellen Wagner, then an associate professor of Educational Technology at the 
University of Northern Colorado, published a list of 18 factors which, in Wagner’s 
words: 

...represent a broad look at issues which impact on distance education success, 
whether one is working in post-secondary, K-12, or corporate training environments 
(Wagner, 1993, p. 28). 

By 1995, Wagner had reviewed and refined this list; changing several variables and 
eliminating others. The list now identifies a total of 16 variables which Wagner suggests 
need to be considered if a distance learning program is to be successful. 

While advances in the application of technology have impacted on the practice of 
distance learning since 1995, the list still serves as a guide to those issues which demand 
consideration if the distance learning endeavor is to be successful. 



In using the list, Wagner acknowledges that the identified variables need to be applied 
against a “continuum of issues” (Wagner, 1995, p. 18). She suggests that these issues are 
situation specific and must be viewed in the context of the particular distance learning 
environment. They range, she argues, from those concerning the simple classroom 
application of distance learning programs to those that are more applicable to the 
corporate boardroom. She also suggests that in reviewing the factors, not all situations 
can be anticipated. The value of the list, she argues, is in the way it is used to guide and 
plan the distance learning activities. 

The list of variables identified by Wagner and a short justification for inclusion on the list 
is shown below. 

Needs assessment. A needs assessment should be conducted to ensure that the distance 
learning endeavor is targeted correctly. 

Audience analysis. Knowledge of the intended audience will assist in preparing the most 
appropriate strategy. 

Instructional design. A systematic design of the distance learning program will assist in 
ensuring that the training outcome is suitable for the intended purpose. 

Course selection. This may include demand for the course, instructor availability, course 
uniqueness and/or course adaptability. Courses not suitable for distance delivery are 
unlikely to succeed. 

Course reconfiguration. If the course intended for distance delivery is an existing one, is 
it’s design suitable for distance learning? 

Lesson planning. In planning lessons for distance delivery, six elements are needed: 
knowledge of the learners; a starting point; clear instructional objectives; a vision of the 
content; practice activities; and an ending point. 




Interactive instructional strategies. Active learner participation is critical to the success 
of distance learning programs. 



Instructional delivery systems. The means of delivery must be complete, appropriate and 
maintained. 

Adapting media and materials for distance delivery. A variety of learning styles should 
be addressed through the creative use of media. 

Developing effective graphics. Any graphics used in the delivery of the program need to 
be designed with the delivery medium in mind. 

Using facilitators at distance education sites. Competent facilitators who understand the 
training objectives and the content will assist the distance learning initiative. 

Learner and instructor support services. 

The success of a distance learning experience is often determined by the quality of the 
services that support the educational process (Wagner, 1995, p. 19). 

Program evaluation and learner assessment. Evaluation, including learner assessment, 
should seek to improve both the processes and product of the activity. 

Organizational readiness. As an agent of change, any distance learning program must 
seek organizational support from all levels of the corporate ladder. 

Inter-institutional partnerships. The open exchange of information, expertise and ideas 
will facilitate the success of a distance learning program. 

Regulatory environment. The environment into which a distance education program is 
implemented will have rules and regulations. An understanding of these will increase the 
likelihood that the activity will be a success. 

Method 

It is intended to test the validity of these factors by reviewing a number of case studies 
and journal articles that describe distance learning activities and/or success factors. The 
case studies and journal articles all relate to organizations that have implemented a 
distance learning course or program and where the outcome has been positive. 
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In an attempt to limit the variables impacting on this study, however, not all case studies 
were accepted for review. The criteria against which case studies were selected are as 
follows. 

• Only case studies which reported the use of electronic media were used (simply in 
an attempt to reduce the scope of this study and to ensure that the case studies are 
appropriate to the future of distance learning).. 

• Case studies more than five years old were excluded. 

• All selected case studies were to provide some degree of description of both the 
planning and implementation of the identified distance learning activity. 

Each of the case studies were reviewed against the factors detailed in Wagner (1995) and 
an assessment was made regarding whether there was enough evidence to suggest that 
these factors were taken into consideration during the design or implementation of the 
course. 

Questions were developed to guide the interpretation of the case studies in relating them 
to Wagner’s list of factors. The guiding questions for each of the factors is shown below. 

Needs assessment. Is there evidence to suggest that a needs assessment was conducted 
and used to guide the development or redesign of the course? 

Audience analysis. Was an audience analysis conducted either as part of the needs 
assessment or as an independent analysis? 

Instructional design. Is there evidence to suggest that a systematic instructional design 
method was used in designing or redesigning the course? 

Course selection. Was the course selected against criteria and assessed as being suitable 
for delivery by a distance learning medium? 

Course reconfiguration. For existing courses, was the courses specifically redesigned for 
distance delivery? 
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Lesson planning. Were lessons planned specifically with a distance delivery mode in 
mind? 

Interactive instructional strategies. Was interactivity considered to be an important 
element in the implementation of the course? 

Instructional delivery systems. Was there any evidence to suggest that an analysis of the 
delivery system was conducted to ensure that it was suitable and readily available for 
both the delivery institution and the learners? 

Adapting media and materials for distance delivery. Were there various adaptations to 
the media and materials to ensure that a variety of learning styles were accommodated? 

Developing effective graphics. Was there any evidence to suggest that the graphics used 
were suitable for the particular delivery media? 

Using facilitators at distance education sites. Were facilitators or tutors used in a face to 
face role with the learners? 

Learner and instructor support services. Was support available to the learners and/or the 
instructors to assist in learning or instructing by distance means? 

Program evaluation and learner assessment. Is there evidence to suggest that the course 
was evaluated or that learner assessment results were used to assist with the evaluation of 
the course? 

Organizational readiness. Is there evidence to suggest that the course was implemented 
with the support of the organization? 

Inter-institutional partnerships. Was the course a collaborative effort between two or 
more organizations where information sharing provided a synergist impact on the 
outcome? 

Regulatory environment. Were the limitations or opportunities associated with the 
regulatory environment considered during the design and implementation of the courses? 
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For each positive answer to the above questions, one score was awarded against that 
particular factor. These were then tallied and the results analyzed in terms of: 

• the number of times each factor was identified in the literature; and 

• the nature of the context in which the program was implemented compared to the 
type of media used. 

In addition to the case studies, a small number of other articles were included in the 
analysis. These articles were chosen due to the nature of the data that they presented and 
the similarity of their aim to the aim of Wagner (1995). The purpose of including these 
articles was simply to address a potential imbalance that the reporting of the case studies 
may induce. Stated in another way, some factors may not have been reported due to the 
obscure nature of the factor itself; not because it was not considered during design or 
implementation. For example, while the regulatory environment may be an important 
issue and was considered (even in a subliminal sense), it may not receive much 
recognition in the cases studies. 

The absence of evidence to suggest that the issue was considered may simply be due to 
the reporting process rather than the events as they occurred. This is not a reflection on 
the quality of the case study reports but simply a product of the purpose of the reports. 

The inclusion of journal articles was designed to counter the argument that if a particular 
factor was not reported in the case study it does not mean that it was not considered. 
Inclusion of ‘lessons leaned’, ‘hints and tips’ or other relevant articles will balance this 
potential reporting inadequacy. 



Assumptions and limitations 

It is assumed that all case studies reviewed were successful. While none of the case 
studies reported empirical data showing that the course was successful, most indicated 
that they were successful at least as perceived by the author(s). This assumption is also 
considered reasonable as most case studies are reported by individuals intimately 
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involved in the planning and/or implementation of the particular course. As a result, only 
successful activities are likely to be reported. 

The validity of this study is limited by the data collection methods. A review of the 
literature may not be a true representation of the actual events. A more detailed analysis 
of the case studies (i.e., an analysis based on data collection methods other than literature 
reviews) may reveal that more factors than those identified in the paper were assessed. 
The factors addressed in the literature review, for example, may only be those which are 
obvious and/or easy to measure. Still others may have been addressed without the 
knowledge of the designer or Project Manager (e.g., as a routine activity of the 
organization). 

It should also be noted that the methodology only allows for a score of either one or zero 
to be allocated. This does not recognize the importance the case study or article places 
on one particular factor when compared with another. In any given article or case study, 
a single point is awarded regardless of whether a factor is mentioned just once or a dozen 
times. The relative importance placed on each factor by the case studies was not 
measured. 

One final limitation is the small number of case studies and articles that were reviewed. 

A larger number would increase the validity of the data. 

Results 

The results of the literature review revealed that evidence was available to suggest that all 
of the factors identified by Wagner (1995) were addressed at some stage during the 
course design or implementation. 

A total of nine case studies and three non-case study articles were reviewed. They 
represented a wide range of settings and media (within the defined limits). A breakdown 
of the number of case studies and articles reviewed (by setting and media type) is shown 
in Table 1 . Each of the non-case study articles are indicated with an star (*). One star for 
each non case study. 
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Higher 

education 


Business 

and 

industry 


K-12 


No setting 
indicated 


Total 


Videoconferencing 


1 




1 


1 


3 


CD ROM 




1 






1 


Computer 

conferencing 

(Asynchronous) 


1 








1 


Internet/Intranet 


1 






l* 


2* 


Talkback TV 


2 








2 


Teleconferencing 


1 








1 


Not indicated 








2 ** 


2** 


Total 


6 


1 


1 


4 


12 



Table l - Number of case studies and articles reviewed (by setting and media type). 
A * indicates a non-case study article. Two * 's indicate two non-case study articles. 



Table 2 shows the number of articles in which evidence was found to indicate that the 
factors on Wagner’s list had been considered during the design or implementation of the 
course. It also shows the breakdown by factor for both the case studies and the non-case 
study articles. Finally, Table 2 shows the mean, high score, low score, standard deviation 
and the number of factors that fall within one standard deviation from the mean. 
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Frequency 




Factors 


Case studies 
n = 9 


Other 
articles 
n = 3 


Total 
n = 12 


Needs assessment 


3 


1 


4 


Audience analysis 


5 


2 


7 


Instructional design 


2 


3 


5 


Course selection 


4 


0 


4 


Course reconfiguration 


1 


1 


4 


Lesson planning 


3 


0 


3 


Interactive instructional strategies 


6 


3 


9 


Instructional delivery systems 


5 


0 


5 


Adapting media and materials for distance 
delivery 


7 


3 


10 


Developing effective graphics 


2 


1 


3 


Using facilitators at distance education sites 


3 


1 


4 


Learner and instructor support services 


6 


1 


7 


Program evaluation and learner assessment 


5 


1 


6 


Organizational readiness 


4 


1 


5 


Inter-institutional partnerships 


4 


0 


4 


Regulatory environment 


2 


1 


3 


Mean 


4.4 


1.2 


5.9 


(mean as a % of n) 


(49%) 


(40%) 


(49%) 


Highest score 


7 


3 


10 


Lowest score 


1 


0 


3 


Standard deviation 


1.7 


Not calculated 


1.9 


Number of factors tailing within 1 standard 
1 deviation of the mean 


ii 

(68%) 


Not calculated 


10 

(62%) 



Table 2 - Number of articles in which evidence was found to suggest that the factors had been 
considered during the design or implementation of the course. 



In addition to the factors identified by Wagner, analysis of the case studies revealed that a 
number of other factors were also considered to be important in the success of the 
surveyed case studies and articles. While collection of this data was not identified in the 
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proposed methodology, they represent a good foundation upon which further research 
may be based. These findings are shown in Table 3. 



Additional factors 


Frequency 


Case studies 
n = 9 


Other 
articles 
n = 3 


Total 
n = 12 


Understanding of technology by instructors, tutors 
and learners 


3 


0 


3 


Small number of learners per instructor or tutor 


1 


0 


1 


Adequate budget and resources available 


1 


2 


3 


Suitability of technology (technically, to 
objectives of course) 


2 


1 


3 


Sound marketing tactics 


2 


0 


2 


Simple administrative procedures 


1 


0 


1 


Good project management skills 


i 


1 


2 


Being learner centered 


2 


0 


2 


Availability of resources 


1 


0 


i 


Total 


14 


4 


18 



Table 3- Other factors considered to be important in the success of distance learning programs 



Discussion 

Highest and lowest scores 

The results described above reveal several trends despite the limited number of case 
studies and journal articles reviewed. The most significant being that evidence was found 
to indicate that all factors were considered at least three times throughout the twelve 
articles (25% of the reviewed literature); although all factors were not considered by any 
single article. This suggests that the factors identified by Wagner (1995) are recognized 
as variable which warrant consideration. 





The lowest scoring factors were: ‘lesson planning’, ‘developing effective graphics’ and 
‘regulatory environment’. However, this does not mean that these factors were not 
considered during the design or implementation of the course. Instead, it simply may 
indicate that the factors were not reported (in either a direct or indirect way) as being 
considered. It is unlikely, for example, that lessons were prepared without some form of 
lesson plan. 

The lack of reporting of these factors is not a reflection on the quality of the case study 
reports, but rather a reflection of the fact that the case studies were simply recording the 
success of a particular course delivered by distance means. They were not seeking to 
measure success against the Wagner variables. 

The highest scoring factor was ‘Adapting media and materials for distance delivery’ 
(identified by ten articles or 83% of the reviewed literature), followed closely by 
‘Interactive instructional strategies’ (identified by 9 articles or 75% of the reviewed 
literature). Both indicate the importance placed on using the media to its fullest and 
seeking active learner participation in the course. 

The interactivity issue was stressed in many of the case studies. Although the relative 
importance of each factor was not measured, it became clear that interactivity was 
considered a key element in successful distance learning programs. 



Setting versus media type 

Table 1 shows the number of articles reviewed by setting and media type. The limited 
number of case studies and other journal articles reviewed does not allow much 
meaningful interpretation of the data, however, each cell within this table has the 
potential to be explored further using the Wagner criteria as a basis for an analysis. If 
more data was collected, this type of analysis would potentially reveal trends in the type 
of variables each setting and delivery medium depends upon for success. 



Assuming the validity of the analysis could be confirmed, the value of this information 
would be substantial. In the context of this study, however, analysis of this type was not 
possible due to the shortage of data. 



Additional factors 

The study also revealed that there was a wide range of additional factors that warrant 
consideration during the design and/or implementation of a distance learning program. A 
total of nine additional variables were identified. The most common factors - 
‘understanding technology’, ‘budget’ and ‘suitability of the technology’ - were, on 
occasion, strongly emphasized in the literature. This indicates that there is a case for 
Wagner’s list to be expanded to include some of these additional factors. 

It should also be noted, however, that the identification of these additional factors relied 
on the subjectivity of the researcher and was largely dependent on the semantics of the 
case study sentence structure. As a result, several of the factors may in fact have been 
better placed within one of Wagner’s factors. 



Non-case studies 

While the inclusion of the non-case study articles was aimed at leveling the imbalance 
that may have been present due to the nature of the case study information, it was shown 
to be an unnecessary precaution. When looked at in relative terms, the results indicate 
that there was no significant difference between the case study (only) scores and the total 
scores. The mean score for these sets of data, for example, were (relative to the number 
in the sample), exactly the same. 

This indicates that the inclusion of non-case study articles aimed at proactively 
addressing this potential for imbalance was not required. Any future studies of this 
nature should review case studies only. 
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Generic variables 

There were a further nine factors identified which were not included in Wagner’s list. 
This indicates that the range and scope of variables that contribute to the success of 
distance learning programs is considerable. 

The factors identified by Wagner (1995) are valid indicators of the issues which 
successful distance learning programs should consider during the design or 
implementation of the program. But the relative importance of each factor will depend 
upon the individual circumstance associated with the particular program. For example, it 
is reasonable to assume, based on the findings of this study, that the factors which 
warrant consideration for a K-12 program which uses videoconferencing will have a 
different variable emphasis than a business and industry course delivered over the 
internet. 

The conclusion drawn from this is that the determination of a generic and valid set of 
critical variables is going to present a significant challenge to distance learning 
professionals interested in this field of research. 



Conclusion 



The findings from this study indicate that the factors identified by Wagner (1995) are 
generally considered when designing of implementing distance learning programs. The 
suitability of the factors, however, is dependent on the nature of the organization into 
which the program is being delivered, and the selection of delivery mode. Nevertheless, 
the findings confirm the assertions by Wagner that: 



General recommendations cannot anticipate every situation that may arise in specific 
distance education applications. Nevertheless, they can serve as benchmarks to guide 
the planning, development, implementation, and evaluation of distance education 
systems, programs, courses, and services (Wagner, 1995, p. 18). 

This assertion of Wagner’s, however, is grounded upon the existing body of knowledge 
regarding the fledgling distance learning industry. Perhaps at some distance point in the 
future, general recommendations which have been based on sound research and 
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evaluation, may be able to anticipate the full range of situations that may arise. The field 
is young; who knows what may be possible. 

There is considerable value in pursuing further research in order to develop a generic set 
of factors which, if considered, will add to the likelihood of success of distance learning 
projects. The potential outcome is a key set of factors that are able to be used with 
confidence by any distance learning project manager. 

The range of variables across the scope of possible distance learning alternatives and 
settings, however, makes it difficult to rationalize just one set of generic factors. Instead, 
a number of variables, ranked in order of importance, may prove easier to develop and 
more useful until the research reaches a greater level of maturity. In order to achieve this, 
however (using the methodology employed by this study), a greater number of cases will 
need to be reviewed and each one, ideally, should be articulated in terms of the variables 
being analyzed. 

Further research in this area is recommended. The identification of variables which 
impact on the success of distance learning projects will provide a foundation upon which 
to commence identifying a quantifiable, generic and valid set of critical success factors 
for the implementation of distance learning programs. In an attempt to reach this goal the 
following are offered as potential future research questions. 

• What are the variables that impact on the success of distance learning programs in 
various settings and when using different types of media? 

• Can a plan be developed to guide research towards developing a comprehensive, 
yet functional list of variables to assist in successfully designing and implementing 
distance learning projects? If so, how can it be implemented to coordinate 
disparate research efforts? 

• Is there a format in which case studies can be presented which will facilitate their 
use in further research? 
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• Is it possible to prioritize the variables that impact on distance learning in terms of 
importance? Should this be done for each distance learning medium and 
accommodate the range of distance learning environments? 

While these questions represent only a starting point for further research, they also 
expose the need for a considerable research investment in distance learning. More 
importantly, however, they represent a starting point for cementing distance learning as a 
competitive business tool. The potential deserves exploration. 
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